The invention relates to a method, a head-up display and an output system for the perspective transformation and outputting, in particular display, of a rendered and/or captured image content, and to a vehicle comprising such a head-up display or output system.
Vehicles, in particular motor vehicles, generally comprise devices for displaying information. The information can both concern the operating state of the vehicle, for example the tank filling level, the speed, the status of the lighting system, etc., and facilitate operation of the vehicle for the driver, for example by displaying navigation information, displaying parking information, etc. In particular, it is possible to capture the environment of the vehicle in a sensor-based manner, for instance by means of one or more cameras, and to represent this vehicle environment for example as a plan view on a display in the vehicle. In order to display the information to the driver as conveniently as possible, head-up displays are used, which project the information to be displayed into the driver's natural field of view, where the driver perceives it visually as a virtual image. In this case, the natural field of view corresponds to the vehicle environment that is primarily to be taken into consideration for the operation of the vehicle, in particular the course of the road ahead of the vehicle. When the information is projected onto the windshield of the vehicle, for example, the driver can read off the information without having to shift his/her gaze from the road, which increases safety when using the vehicle.
By virtue of curvatures of optical elements of the head-up display and the windshield and the fact that the driver generally looks at the windshield at an angle, the image of the information to be displayed is distorted. Equally, however, a vehicle environment captured in a sensor-based manner, in particular by means of one or more cameras, is also distorted by curvatures of imaging elements of a capturing sensor device, for instance by one or more cameras using a fish-eye lens. In order to represent the information to the driver in a desired manner, in particular without distortion, i.e. clearly and readily recognizably, the imaging of the information to be displayed, in particular of the vehicle environment, therefore has to be adapted to the beam path from the image-generating device of the head-up display via optical elements of the head-up display and the windshield of the vehicle to the driver's eye, on the one hand, or to the imaging properties of an image-generating device, on the other hand.
For adaptation to the beam path or the imaging properties of the image-generating device, the information to be displayed is transferred as rendered or captured image content to a graphics module (graphics engine), which subjects the rendered or captured image content to a transformation. The transformation is designed to compensate for the distortion that arises for instance as a result of curved, in particular optical, elements of the head-up display or of the image-generating device, a curved windshield and/or a non-perpendicular viewing angle with respect to the windshield. In other words, the rendered image content is distorted by the transformation of the graphics module in order to enable the observer, in particular the driver, to visually perceive a rectified image of the rendered image content. Accordingly, however, an image content that has been captured, in particular in a sensor-based manner, can also be conditioned by the transformation in such a way that it can be perceived by the driver in a visually rectified manner, in particular directly, on an image-generating device.
It is an object of the invention to specify a method, a head-up display, a display system and a vehicle in which the transformation and outputting, in particular display, of a rendered image content and/or image content captured in a sensor-based manner is carried out more flexibly and more rapidly.
It is a further object of the invention to adapt a transformation that transforms a rendered image content more easily to changes in a beam path of a head-up display, caused for example by a change in the position of an image of the rendered image content on a projection surface of the head-up display.
It is a further object of the invention to adapt a transformation that transforms a captured image content more easily to imaging properties of an image-generating device, particularly if the image content captured in a sensor-based manner would be reproduced in a distorted manner on the image-generating device.
A first aspect of the invention relates to a method for the perspective transformation and display of a rendered image content by a head-up display, wherein the rendered image content, i.e. image content to be displayed, is divided into a plurality of regions, preferably tiles or rectangles. The tiles, or the image content contained in the individual tiles, are or is preferably transformed in each case by means of a perspective transformation and the individual, perspectively transformed tiles, or perspectively transformed image contents contained in the perspectively transformed tiles, are joined together to form a perspectively transformed image content. The perspectively transformed image content is projected onto a projection surface assigned to the head-up display.
A second aspect of the invention relates to a method for the perspective transformation and outputting of an image content captured, in particular in a sensor-based manner, for instance by means of one or more cameras, wherein the image content that has been captured, and in particular is to be displayed, is divided into a plurality of regions, preferably tiles or rectangles. The tiles, or the image content contained in the individual tiles, are or is preferably transformed in each case by means of a perspective transformation and the individual, perspectively transformed tiles, or perspectively transformed image contents contained in the perspectively transformed tiles, are joined together to form a perspectively transformed image content. The perspectively transformed image content is output.
A third aspect of the invention relates to a head-up display for the perspective transformation and display of a rendered image content, which head-up display comprises a control unit and a projection unit. In particular, the head-up display according to the invention is configured to carry out the perspective transformation and display of a rendered image content according to the first aspect of the invention. Preferably, the control unit comprises a first module, configured for dividing the rendered image content, i.e. image content to be displayed, into a plurality of regions, preferably tiles or rectangles, a second module, in particular a graphics module, configured for transforming the individual tiles, or the image content contained in the individual tiles, by means of a respective perspective transformation, in particular via a graphics programming interface, and a third module, configured for combining the individual, perspectively transformed tiles to form a transformed image content. Furthermore, the projection unit is preferably configured for projecting the perspectively transformed image content onto a projection surface assigned to the head-up display.
A fourth aspect of the invention relates to an output system for the perspective transformation and outputting of an image content captured, in particular in a sensor-based manner, for instance by means of one or more cameras, which output system comprises a control unit and an output unit, in particular data interface. In particular, the output system according to the invention is configured to carry out the perspective transformation and outputting of a rendered image content according to the second aspect of the invention. Preferably, the control unit comprises a first module, configured for dividing the captured image content, in particular image content to be displayed, into a plurality of regions, preferably tiles or rectangles, a second module, in particular a graphics module, configured for transforming the individual tiles, or the image content contained in the individual tiles, by means of a respective perspective transformation, in particular via a graphics programming interface, and a third module, configured for combining the individual, perspectively transformed tiles to form a transformed image content. Furthermore, the output unit is configured for outputting the perspectively transformed image content.
The first, second and third modules can be embodied in each case as hardware or software module. In particular, one or two modules can be embodied as software module and the remaining module(s) can be embodied as hardware module.
Preferably, a first, second or third module embodied as software module is implemented on one or more processors, in particular central processing unit or central processing units (CPU) of the control unit.
Preferably, a first, second or third module embodied as hardware module comprises at least one graphics card having a graphics processing unit (GPU).
A fifth aspect of the invention relates to a vehicle comprising a head-up display in accordance with the third aspect of the invention or a display system in accordance with the fourth aspect of the invention.
The features and advantages described with regard to the first aspect of the invention and the advantageous configuration thereof are also applicable to the second, third, fourth and fifth aspects of the invention and the advantageous configurations thereof, and vice versa.
An image content within the meaning of the invention should be understood to be, in particular, the optical image of one or more objects. By way of example, an object represented in an image is an image content (for instance a tree). While the image changes under a perspective transformation, i.e. becomes a different image, the image content however does not change (the changed image still shows the, possibly distorted, tree).
Capture within the meaning of the invention is carried out by means of a sensor device, in particular by means of one or more optical sensors. Preferably, in this case, one or more objects, for example a vehicle environment, is or are recorded. In this case, the object(s) can also be recorded (filmed) sequentially, that is to say substantially continuously. In particular, a captured object can be represented or optically imaged by means of a display, for instance a display unit or a projection unit.
A tile within the meaning of the invention should be understood to be, in particular, a portion or region of an image or of an image content. Preferably, a tile of an image or image content comprises a portion of the image content. Preferably, a tile has a rectangular shape. Alternatively, however, a tile can also have a different shape, for instance be embodied as a trapezoid, a parallelogram, a rhombus or a triangle. In particular, an image or image content can be decomposed or divided into tiles without any gaps.
Outputting within the meaning of the invention should be understood to be providing perspectively transformed image content. Preferably, the image content in this case is provided to a display unit for visual representation. Alternatively or additionally, the image content is provided as data, in particular via a data interface. Preferably, the image content in this case is provided to a driver assistance device, preferably for deriving control parameters for highly automatic or semiautomatic, in particular autonomous, driving.
A display unit within the meaning of the invention should be understood to be, in particular, an imaging device which can represent or display an image or image information. By way of example, a display unit can be embodied at least partly as a display or screen.
Within the meaning of the invention, the imaging properties of a display unit or of an image-generating device should be understood in particular always with respect to imaging properties of a sensor device that captures an image content, for instance of one or more cameras and/or lenses. By way of example, an image content captured by a fish-eye lens would be represented without distortion on a correspondingly curved display unit. In the case of a planar display unit, by contrast, a corresponding perspective transformation of the image content would be necessary in order to represent the image content without distortion on the display unit.
The invention is based in particular on the approach of dividing an image to be displayed, in particular a rendered or captured image content, which contains information to be displayed, into a plurality of regions, for example tiles, preferably rectangles or rectangular tiles, and transforming said regions separately by a respective perspective transformation. In other words, the perspective transformation of the image, i.e. of the rendered or captured image content, is carried out modularly. Afterward, the perspectively transformed regions, i.e. tiles, are combined again to form a now perspectively transformed image or image content, which, in or by a head-up display, is projected onto a projection surface or is output to an output unit, in particular displayed on a display unit, and can be visually perceived by an observer, in particular a driver, for instance in a vehicle comprising a head-up display, preferably without distortion, i.e. clearly and readily recognizably, on account of the perspective transformation performed.
In this case, the perspective transformation of individual tiles is preferably coordinated with the respective individual tile to be perspectively transformed. If a change in the beam path of the head-up display necessitates the adaptation of the image to be displayed, i.e. of the rendered image content, this can be brought about by coordination or adaption of individual perspective transformations.
Likewise, a change in the imaging properties of a sensor device that captures the image content, caused for example by the use of a different lens in front of a camera, as a result of which a perspectively transformed image content would be represented in a distorted manner by the display unit, can be compensated for by coordination or adaptation of individual perspective transformations.
In particular, the captured image content can be transformed in such a way that the driver can easily visually register and/or assess the perspectively transformed image content represented on the display unit. Preferably, the perspectively transformed image content can also be represented particularly memorably as a result.
In particular, it is not necessary to coordinate or to adapt each individual one of the perspective transformations, or to perform an adaptation, i.e. change, of a single global perspective transformation that transforms the entire image to be displayed, i.e. the entire rendered or captured image content. By comparison with the adaptation of each individual perspective transformation or a single global perspective transformation, the method according to the invention requires a lower computing power and therefore increases the frame rate at which the transformed images can be displayed by the head-up display or the display unit.
Preferably, the invention makes it possible to dispense with the perspective transformation of individual tiles if this is not necessary for a distortion-free representation or display of the combined and transformed image, i.e. image content. Particularly if only a portion of the image to be displayed, i.e. of the rendered or captured image content, has to be distorted or rectified by means of a perspective transformation or transformation for adaptation to the beam path of the head-up display or the observer's position, on the one hand, or to an imaging property of the display unit, on the other hand, by means of the modular perspective transformation it is possible to reduce the computational complexity for perspective distortion or rectification by not transforming individual tiles whose image content does not have to be adapted to the beam path or the imaging property.
Preferably, the perspective transformation of individual tiles, i.e. a modular perspective transformation, furthermore allows a memory- and computation-time-saving perspective transformation by means of a graphics module, which is preferably in widespread use and runs stably in particular on current microprocessors, in particular on different platforms. With further preference, said graphics module is designed to process vector graphics in particular with hardware acceleration, such that it is possible to carry out individual perspective transformations on a vector basis particularly rapidly and efficiently.
Overall, the invention enables rapid and flexible transformation and display of a rendered or captured image content.
Preferably, the perspectively transformed image content can be output to a driver assistance device and be used for determining control parameters for highly automatic or semiautomatic, in particular autonomous driving. In this case, the image content is preferably processed by image recognition algorithms known from the prior art, wherein the image content is preferably perspectively transformed in such a way that said algorithms can be implemented particularly reliably and/or efficiently.
By way of example, it is possible, on the basis of the perspectively transformed image content, to ensure, in particular directly in the vicinity of the vehicle, accurate determination of objects or a clearance, taking account of which the vehicle can be controlled reliably, i.e. in a manner free of collisions.
In one advantageous configuration, the perspectively transformed image content is displayed on a display unit. As a result, the image content captured in a sensor-based manner can be visually perceived easily and clearly by a driver, in particular on an arbitrary display.
In a further advantageous configuration, the individual tiles are transformed in such a way that the perspectively transformed image content is represented by the display unit in an uncompressed manner, in particular without distortion, in at least one first region, in particular in a center of the image or of the image content, and in a compressed manner, i.e. in a distorted manner, in at least one second region, in particular at the image edge or at the edge of the image content. As a result, the image content can advantageously be represented nonlinearly, such that, for example, in a plan view of the vehicle and/or the surroundings of the vehicle, even with a large viewing angle, details directly in the vicinity of the vehicle can be represented readily recognizably.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, during the transformation each individual tile is perspectively transformed with regard to at least one of the following features: trapezoid shape, curvature, stretching and/or compression, rotation, offset. In particular, each individual tile can also be perspectively transformed by means of a combination of at least two of these features. A reliable adaptation of the image content to be displayed to the beam path of the head-up display or the imaging properties of the display unit is possible as a result.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, adjoining tiles have overlapping image contents. In particular, the overlapping image contents each have a width of one, two or three pixels. This ensures that the perspectively transformed tiles, during the process of combining to form a perspectively transformed image content, likewise have overlapping, perspectively transformed image content or the edges of the perspectively transformed tiles terminate flush, in particular. In particular, this reliably prevents the combined, perspectively transformed image content from having regions which are not covered by at least one perspectively transformed tile and are therefore represented as color-neutral, in particular black or white, pixels. Preferably, this feature of the method also prevents said regions from arising by means of applying an edge smoothing, in particular by means of a smoothing filter.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the image content is present as raster graphics, wherein individual pixels of a tile are displaced by the perspective transformation. As a result, the rendered or captured image content can be divided particularly easily into different regions, preferably tiles or rectangles or rectangular tiles, wherein in each case a predefined number of pixels of the raster graphics are assigned to a region, or to a tile. The individual pixels of the raster graphics that belong to a respective tile are displaced by the perspective transformation. Consequently, the perspective transformation can be realized particularly easily.
Preferably, the raster graphics have a specific color depth, in particular 32, 24, 16, 8 or 4 bits. With further preference, the raster graphics can also have a color depth that lies between these values. With further preference, the raster graphics can also have a color depth that is greater than 32 bits or less than 4 bits. With further preference, different pixels of the raster graphics each have different color depths. In particular, preferably, some contiguous pixels of the raster graphics which respectively belong to an individual tile have in each case the same color depth. Particularly preferably, the color depth of an individual tile has a magnitude only such as is necessary for the true-color representation of the image content contained in the individual tiles. As a result, in particular, the rendered image content requires particularly little memory space, in particular in a buffer memory, and the perspective transformations of the tiles can be carried out particularly efficiently.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, an interpolation, in particular a linear interpolation, of individual pixels of a tile, in particular of the pixels which adjoin pixels displaced by the perspective transformation and/or lie between said pixels, is carried out during the perspective transformation. This reliably avoids the situation in which regions having pixels to which no color value was assigned arise. Preferably, the number of pixels per tile is low, such that individual pixels are not displaced over large distances during the perspective transformation. In this case, in particular, a linear interpolation of the pixels lying between the pixels displaced during the perspective transformation yields a perspectively transformed image content whose image generated by a head-up display or a display unit can be visually perceived distinctly and clearly, i.e. in particular without artifacts. A linear interpolation causes in particular only a low computational complexity, that is to say that the burden for a calculating processor is low, for which reason the perspective transformation can be carried out rapidly and efficiently.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the individual tiles are perspectively transformed by multiplication by a transformation matrix, which allows the use of vector graphics technology. Preferably, the transformation matrix is a 3×3 matrix. Multiplication by a transformation matrix is a particularly simple and precise way of carrying out a transformation. A corresponding perspective transformation can therefore be carried out particularly rapidly and efficiently, in particular by a graphics module. In particular, it is therefore possible to achieve, for the frame rate at which rendered or captured image contents are perspectively transformed, a value of 30 to 90, preferably of 45 to 75, particularly preferably of substantially 60, perspectively transformed image contents per second. In one particularly advantageous configuration of the method, it is possible to achieve a value of 90 to 160, preferably of substantially 120, perspectively transformed image contents per second.
Preferably, during the matrix multiplication by a transformation matrix, the positions of individual pixels of a tile, in particular of the four corner points, which are preferably given by support vectors, are multiplied by the transformation matrix. In this case, support vectors are vectors which specify the position of a pixel relative to an origin, wherein the origin may be in particular a corner point or midpoint of a tile or of the image, i.e. of the rendered or captured image content. As a result of the multiplication of the support vectors by the transformation matrix, the position of the individual pixels of the tile shifts relative to the origin. Preferably, pixels which lie between these displaced pixels, in particular in a manner adjoining the latter, are interpolated, in particular by a linear interpolation. Preferably, gaps in the image content of the perspectively transformed tile are closed as a result.
Preferably, by the choice of the number and/or the dimensions, i.e. the size or area, of the tiles when dividing the image content to be displayed, the number of pixels to be interpolated linearly, in particular, can be determined, in particular limited, since the number of pixels per tile that have to be perspectively transformed decreases as the number of tiles increases. This ensures that in particular a linear interpolation of pixels that lie between displaced pixels, in particular in a manner adjoining displaced pixels, generates clear and distinct, perspectively transformed image content on the tiles or the combined perspectively transformed image content to be displayed via the head-up display or the display or to be output by the output unit, i.e. perspectively transformed image content without artifacts.
Preferably, the number and/or the dimensions of the tiles are/is chosen to be greater than a critical number and/or size for which the number of pixels to be interpolated linearly, in particular, is so small that artifacts, in particular corners, occur in the perspectively transformed image content. This ensures that the combined, perspectively transformed image content to be displayed via the head-up display or the display unit or to be output by the output unit has no artifacts, in particular corners. Preferably, the number and/or size of the tiles are/is chosen such that the tiles overlap, in particular in each case with one or two pixels.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the transformation matrix is chosen from a multiplicity of transformation matrices. In particular, the multiplicity of transformation matrices here preferably includes in each case a transformation matrix for each tile to be perspectively transformed. This can ensure that the transformation matrices are adapted to the beam path of the imaging of the tile to be perspectively transformed in each case in the head-up display, on the one hand, or to the imaging properties of the display unit, in particular with respect to the imaging properties of the sensor device that captures the image content, on the other hand, or can be adapted in particular individually to changes in the beam path of the imaging or the imaging properties of the display unit. With further preference, the multiplicity of transformation matrices includes various perspective transformations.
Any vectorial perspective transformation can be realized by the use of transformation matrices. As a result, the perspective transformation can be adapted particularly well to the beam path or the imaging properties. Particularly preferably, the perspective transformation carried out with a transformation matrix from the multiplicity of transformation matrices relates to at least one of the following features: trapezoid shape, curvature, stretching and/or compression, rotation, offset.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the perspective transformation of the individual tiles is carried out by a graphics module, preferably via a graphics programming interface, in particular OpenVG. The use of a graphics module significantly accelerates the perspective transformation of the individual tiles, in particular by means of hardware acceleration, and reduces the computational burden for a central processing unit (CPU). Preferably, a graphics programming interface is used which is in widespread use and is available as standard on microprocessors. With further preference, the graphics programming interface is available on a plurality of platforms. Furthermore, the graphics programming interface used preferably allows the use of a graphics module which makes available the hardware-accelerated processing or calculation of vector graphics.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the perspectively transformed tiles are stored in a buffer memory. The buffer memory allows particularly fast access to the perspectively transformed tiles stored therein, in particular during the process of combining to form an entire perspectively transformed image, or image content, to be displayed or to be output.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, only a portion of the plurality of tiles is perspectively transformed and the remaining transformed tiles are loaded from the buffer memory for the purpose of combining the transformed image. As a result, the combination of the perspectively transformed image, or image content, to be displayed can be significantly accelerated. In particular, it is thus possible to achieve a frame rate at which rendered or captured image content is perspectively transformed of 30 to 90, preferably of 45 to 75, particularly preferably of substantially 60, perspectively transformed image contents per second. In one particularly advantageous configuration of the method, it is possible to achieve a value of 90 to 160, preferably of substantially 120, perspectively transformed image contents per second.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, only that portion of the plurality of tiles is perspectively transformed which contains elements of the image content which have changed with respect to an image content that was previously to be transformed and to be output, in particular to be displayed and/or to be projected. In particular, the remaining perspectively transformed tiles required for combining the perspectively transformed tiles to form a perspectively transformed image content to be displayed are loaded from the buffer memory. This ensures, in particular, that a processor demanded by the perspective transformations processes only those parts of the rendered or captured image content which differ from the image content that was previously perspectively transformed and output, in particular displayed, i.e. contain changed information to be displayed or a changed representation of the information to be displayed, as a result of which the computing power of said processor is significantly reduced in comparison with carrying out a perspective transformation of all the tiles. As a result, the computing power of such a processor can be allocated particularly efficiently for calculations and a frame rate at which rendered or captured image content is perspectively transformed of 30 to 90, preferably of 45 to 75, particularly preferably of substantially 60, perspectively transformed image contents per second can be achieved. In one particularly advantageous configuration of the method, it is possible to achieve a value of 90 to 160, preferably of substantially 120, perspectively transformed image contents per second.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, different portions of the plurality of tiles have different color depths, in particular 32, 24, 16, 8 and 4 bits. With further preference, different parts of the plurality of tiles can also have a color depth that lies between these values. With further preference, different parts of the plurality of tiles can also have a color depth that is greater than 32 bits or less than 4 bits. In particular, the tiles have in each case the smallest possible color depth for which the respectively contained image content can still be displayed or represented with true color, i.e. with no reduction in color quality. Preferably, the perspective transformations, in particular the preferably linear interpolations of individual pixels, of the plurality of tiles having the different color depths are carried out at the respective color depth of the tile that is to be perspectively transformed. This ensures that the perspective transformation or the perspectively transformed tile requires only little memory space or the preferably linear interpolation requires little computation demand. As a result, the perspective transformations, in particular the preferably linear interpolations of individual pixels, can be carried out particularly rapidly and efficiently.
In a further advantageous configuration of the method for the perspective transformation and outputting, in particular display, of a rendered or captured image content, the plurality of tiles are mirrored by a point or line mirroring during the perspective transformation and are combined to form a perspectively transformed, mirrored image content. As a result, the image orientation of the perspectively transformed image content to be displayed can be adapted to the beam path of the head-up display or to imaging properties of the display unit. In particular, the mirroring makes it possible to dispense with the use of additional optical elements in the head-up display which change, in particular rotate and/or mirror, the image orientation of the perspectively transformed image content for error-free imaging on the projection surface of the head-up display. As a result, in particular, the head-up display can be realized a particularly simple manner.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.